Systems and methods of monitoring network devices

ABSTRACT

Implementations of the disclosed subject matter provide systems and methods of assigning, at a server, a unique identifier to each of a plurality of devices communicatively coupled to one another and the server via a communications network. Each unique identifier may be converted to a device hash key by applying a hash function. A range of device hash keys of the plurality of devices may be split into N approximately equal sectors, where N is a prime number and each sector includes 1/N of the device hash keys of the plurality of the devices. K monitoring workers provided by the server may monitor the plurality of devices in an order based on the respective device hash key, where K is an integer.

BACKGROUND

In present systems, periodic messages are sent from a device to acentral location via a communications network, so that the operationalstatus of the device can be determined. The absence of a message beingreceived by the central location for a predetermined period of time froma particular device indicates a problem with that device, or with thecommunications network. In some systems, polling is used by the centrallocation to periodically request the operational status of each deviceof the network, and a central record is updated based on the pollingresults.

BRIEF SUMMARY

According to an implementation of the disclosed subject matter, a methodis provided that includes assigning, at a server, a unique identifier toeach of a plurality of devices communicatively coupled to one anotherand the server via a communications network. The method may includeconverting, at the server, each unique identifier to a device hash keyby applying a hash function. At the server, a range of device hash keysof the plurality of devices may be split into N approximately equalsectors, where N is a prime number and each sector includes 1/N of thedevice hash keys of the plurality of the devices. The method may includeproviding, at the server, K monitoring workers to monitor the pluralityof devices in an order based on the respective device hash key, where Kis an integer.

According to an implementation of the disclosed subject matter, a systemis provided in that includes a plurality of devices communicativelycoupled to one another via a communications network. The system mayinclude a server, communicatively coupled to the communications network,to assign a unique identifier to each of the plurality of devices,convert each unique identifier to a device hash key by applying a hashfunction, split a range of device hash keys of the plurality of devicesinto N approximately equal sectors, where N is a prime number and eachsector includes 1/N of the device hash keys of the plurality of thedevices, and provide K monitoring workers to monitor the plurality ofdevices in an order based on the respective device hash key, where K isan integer.

According to an implementation of the disclosed subject matter, meansfor monitoring device of a network are provided, including means forassigning a unique identifier to each of a plurality of devicescommunicatively coupled to one another and the server via acommunications network. Means for converting each unique identifier to adevice hash key by applying a hash function may be provided. A range ofdevice hash keys of the plurality of devices may be split into Napproximately equal sectors, where N is a prime number and each sectorincludes 1/N of the device hash keys of the plurality of the devices. Kmonitoring workers may be provided to monitor the plurality of devicesin an order based on the respective device hash key, where K is aninteger.

Additional features, advantages, and embodiments of the disclosedsubject matter may be set forth or apparent from consideration of thefollowing detailed description, drawings, and claims. Moreover, it is tobe understood that both the foregoing summary and the following detaileddescription are illustrative and are intended to provide furtherexplanation without limiting the scope of the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are included to provide a furtherunderstanding of the disclosed subject matter, are incorporated in andconstitute a part of this specification. The drawings also illustrateembodiments of the disclosed subject matter and together with thedetailed description serve to explain the principles of embodiments ofthe disclosed subject matter. No attempt is made to show structuraldetails in more detail than may be necessary for a fundamentalunderstanding of the disclosed subject matter and various ways in whichit may be practiced.

FIGS. 1A-1B show an example method of monitoring devices in acommunications network according to an implementation of the disclosedsubject matter.

FIG. 2 shows a computing device according to an implementation of thedisclosed subject matter.

FIG. 3 shows a network configuration according to an implementation ofthe disclosed subject matter.

DETAILED DESCRIPTION

Devices communicatively coupled to a communications network may beunreliable. Depending on the size of the network, there may be hundreds,thousands, or millions of potentially unreliable devices coupled to thenetwork. It may be desirable for one or more computers, such as aserver, to have up-to-date information of the status of each device ofthe network. For example, a device status may include: healthy,available, busy, inaccessible, or the like.

In implementations of the disclosed subject matter, device statusinformation for one or more devices of a network may be collected at asingle server location, where device status information collection maybe performed by a selected number of workers. Each worker may besoftware, hardware, or a combination thereof. In some implementations,the workers may be generated (e.g., if the worker includes software)and/or assigned (e.g., if the worker includes a hardware device) by theserver. The number of workers allocated to collect device statusinformation may be based on the number of network devices. The number ofnetwork devices may change, as new devices are added to the network, oras devices are removed from the network. This arrangement may providecentralized, large-scale device management, without overwhelming thecentralized server, and/or any of the individual workers collectingdevice status.

The centralized server arrangement may determine overall systemstatistics, and may compare the desired state of each device to itsactual state. Devices communicatively coupled to the network may becomputers, Internet servers, networking hardware, Internet-of-Thingsdevices or nodes, tablets, laptops, mobile phones, smart watches, and/orsmart devices, and/or any other device that may be accessed remotely andprobed for its current state. In some implementations, at least some ofthe devices coupled to the network may be virtual machines (e.g.,running untrusted software) on one or more servers.

Implementations of the disclosed subject matter may address the problemof balancing between communicating with each device of the networkfrequently enough to have a current status of the device, and reducingthe amount of communications between devices to reduce the communicationtraffic and use or network resources. The disclosed arrangements mayavoid creating communication bottlenecks in the network, which typicallyfound at a central server in current systems, by distributing the devicestatus gathering tasks.

For example, in some current systems, periodic check-in messages (e.g.,“heartbeats”) are sent by each device to a central location. Eachmessage contains the current state of a single device. When the centrallocation does not receive a heartbeat from a particular device after apredetermined period of time, the device or the communication medium(e.g., a portion of the communications network) may have a problem. Insuch current systems, a central location receives these messages, anddetermines an up-to-date health of the overall system. This approach mayhandle a certain maximum number of devices (e.g., tens of thousands) ormaximum heartbeat frequency (e.g., once per minute per device). However,the central system becomes a bottleneck that has trouble processing thenumber of heartbeat messages received when, for example, the maximumnumber of devices or maximum heartbeat frequency is reached or exceeded.Moreover, if the central server is unreliable, such current systems willbe prone to outages.

Another approach used in current systems is polling, where the centrallocation (e.g., a central health monitoring service or the like)periodically issues a request for each device of a network for itscurrent state, and updates a central record based on the received stateinformation. This system avoids the problem of having a single nodeoverloaded with incoming messages (i.e., fan-in). The disadvantage ofthis system is that the central location may be overloaded with sendingoutgoing messages (i.e., fan-out). That is, the system typically cannotsend out enough status check requests from a central location to monitormillions of devices. Another disadvantage of this arrangement is thatfailure of the central location will disable the entire system (i.e., asingle-point-of-failure problem). Thus, the central location must beover-engineered to minimize failure, which may be expensive.

Implementations of the disclosed subject matter may split a datacollection into interchangeable parts, which may increase overall systemreliability and consistency. The disclosed arrangement may usedistributed workers to handle the load of gathering device statusinformation of network devices, which may be stored at the centralserver. That is, unlike current systems, implementations of thedisclosed subject matter do not suffer from excessive fan-in, fan-out,or single-point failure.

FIGS. 1A-1B show an example method 100 of monitoring devices in acommunications network according to an embodiment of the disclosedsubject matter. At operation 102, a server (e.g., server 13 and/orremote platform 17 shown in FIG. 3) may assign a unique identifier toeach of a plurality of devices (e.g., device 10, 11 shown in FIGS. 2-3)communicatively coupled to one another and the server via acommunications network (e.g., network 7 shown in FIG. 3). Each devicecommunicatively coupled to the network may have a globally uniqueidentifier or name, which may be assigned by the server.

Each device that is communicatively coupled to the communicationsnetwork may be configured to receive queries regarding the operatingstate of the device. The queries may be received, for example, vianetworking protocols, such as HTTP (hypertext transfer protocol), TCP/IP(transfer control protocol/internet protocol), and the like. In someimplementations, a proxy service may be used to maintain persistentcommunication channels with each of the devices coupled to the network.The proxy service may use a plurality of workers, which are discussed indetail below, to determine the status of one or more devices of thenetwork. A computer or server, such as server 13 and/or remote platform17 shown in FIG. 3, may provide the proxy service. In someimplementations, the workers may be controlled by the server of theproxy service. This service may determine, for example, that a device ishealthy whenever the corresponding communication connection to thedevice via the network is open (i.e., operational).

At operation 104, each identifier and/or name may be converted by theserver (e.g., server 13 and/or remote platform 17 shown in FIG. 3) intoan integer called a device hash key by applying a hash function. In someimplementations, a hash function such as MD5, SHA-1, or the like may beused. In some implementations, the device hash key may be 64 bits inlength, or any other suitable bit length.

At operation 106, the server may split a range of device hash keys ofthe plurality of devices into N approximately equal sectors (i.e.,sub-ranges), where N may be a prime number and each sector includes 1/Nof the device hash keys of the plurality of the devices. In someinstances, the number of hash keys may not be exactly divisible by N, sosome sectors may be one key larger than other sectors. That is, somesectors may have the 1/N fraction of devices rounded down, while othersectors may have the 1/N fraction of devices rounded up. In someimplementations, the value of N may be 101, or any other suitable primenumber. Selecting a large prime number for N (e.g., 101) may providethat no two workers of the proxy service operated by the server everread the same sector from a database (e.g., database 15 shown in FIG. 3)at the same time, or write a state update to the database for the samedevice, as discussed in detail below.

A plurality of K monitoring workers, where K is an integer, maybedeployed, generated, and/or assigned by the server to monitor theplurality of devices communicatively coupled to the network. As shown inFIG. 1A, the server may provide K monitoring workers to monitor theplurality of devices in an order based on the respective device hash keyat operation 108. In some implementations, K may have a value of 9, ormay have any suitable integer value. Each worker may be software,hardware, or a combination thereof that may be operated continuously.

As shown in FIG. 1B, example method 100 may include performing, by atleast one of the K monitoring workers, a status check of one or moredevices included in at least one of the N sectors at operation 110. Eachworker may monitor the state of each device, one by one, in order oftheir hash keys. If a device is temporarily inaccessible orcommunication with the device via the network is below a predetermineddata rate, a worker may initiate the state check of the next devicebefore the state check of the current one is completed. A status checkof the next device in the one of the N sectors may be initiated beforethe completion of a status check of a current device of the plurality ofdevices when it is determined that the current device is inaccessiblevia the communications network or responding below a predeterminedthreshold.

That is, a worker may determine the operating state of a plurality ofdevices. In some implementations, the operating state of a plurality ofdevices may be determined simultaneously. The system may include acentral database (e.g., database 15 shown in FIG. 3) that records themost recently determined operating state of each device.

A database system (e.g., database 15 shown in FIG. 3) may store thedetermined state of the one or more devices at operation 120. Thedetermined state of each of the plurality of devices may be stored tothe database system when the determined state has changed from thepreviously-stored state. That is, in implementations of the disclosedsubject matter, the database of the system may have the most recentlyverified operating state of each device stored. Workers may not writedevice status data to the database unless a currently determinedoperational state of the device has changed from the operating staterecorded in the database.

In some implementations, each worker may processes one sector of hashkeys at a time. The worker may read data for a sector from the database,and may query the devices in the sector to determine the currentoperating state of each device. The worker may write any determinedoperating state changes to the database. Each sector may includeapproximately 1/N fraction of all devices for the network. The value ofN may be selected to restrict the frequency of database reads (i.e.,requests received by the database to retrieve data) to a predeterminedrate. This rate may be when the database may read and provide therequested data, and not be overwhelmed with requests so as to result ina delay greater than a predetermined amount of time delay.

The frequency of write operations to the database (e.g., requests towrite data to the database) may be based on the number of actual statechanges of devices coupled to the network. In implementations of thedisclosed subject matter, each worker may determine which sector toprocess at a particular point in time. In an illustrative example, thedevice hash keys may be 64-bit integers. That is, the integers of thedevice hash keys may be between 0 and 2 to the power of 64 minus 1,inclusive (i.e., 0 to 2⁶⁴−1)

The range of devices having device hash key numbers may be split into Nsectors, numbered from 0 to N−1, with the j^(th) sector starting at hashkey (2**64)/N*j+min(j,(2**64)% N) and may include (2**64)/N+(j<(2**64)%N)) hash keys. In this equation, ** is the to the power of operator,min(x,y) is the minimum of two integers, the % operator is remainderafter division, the/operator is truncating integer division, and the<operator is less than, which evaluates to 0 or 1.

In implementations of the disclosed subject matter, the difference insize between the largest and smallest sectors may be one (1). In someimplementations, the desired frequency of determining operation statefor each device may be P, which is a measure of time. The system maydetermine the operation state of each device once per P.

In an example, the K workers may be numbered from 0 to K−1. At any giventime t, worker number 0 may process sector number (t % (P*K))*N/(P*K),where operator * is multiplication. In this example, other workers mayoperate in a similar manner to worker number 0, except that they mayadjust their clocks to be (P*i) ahead, where i is the worker number(between 0 and K−1, inclusive). This may ensure that workers are evenlydistributed across the range of device key hashes. This may provide aneven distribution of workload among the workers to determine theoperating state of devices of the network, so that workers are notoverloaded.

The implementations of the disclosed subject matter provides advantagesover present systems that may use heartbeats or centralized polling, asdescribed above. The system of the disclosed subject matter may haveincreased resilience to failures and/or changes. The workers may beallowed to fail (i.e., unable to check status of devices), pause (i.e.,temporarily halt checking the status of a device), and/or restartthemselves without creating an adverse impact on the overall system. Inthe implementations of the disclosed subject matter, a worker failuremay merely double the operating state check period for a set of devices,which may be easily mitigated by decreasing the value of P. An increasein the number of monitored devices may be unlikely to overload a singleworker because hashing will spread the new workload evenly across allworkers. Similarly, a wide-spread operating state change that may affecta large number of devices may be evenly distributed across all workers.

By selected a large prime number for N (e.g., 101), no two workers mayread the same sector from the database at the same time, and/or write anoperational state change for a device at the database for the samedevice.

The system parameters, such as the values for P, K, and/or N, may bechanged without causing instability and/or crashing the whole system(i.e., the system may continue to be operational). The system mayrestart one worker at a time and may allow a temporary inconsistency insystem parameters across workers. The workers may not rely on any kindof centralized control or shared state, besides the database. Theworkers may have synchronized clocks (e.g., not off by more than a fewseconds).

In an example, the server (e.g., server 13 and/or database 15 shown inFIG. 3) may generate, assign, and/or allocate workers to monitor thestatus of gamelets of a cloud-based and/or server-based gaming system(e.g., remote platform 17 shown in FIG. 3). The gamelets may be virtualmachines that are executed on a user device (e.g., device 10, 11 shownin FIGS. 2-3) that are communicatively coupled to a network (e.g.,network 7 shown in FIG. 3). Games executed within gamelets may make thegamelets unstable, such as by overloading the operational capabilitiesof a graphics driver, a kernel controlling the gamelet, a communicationsinterface, or the like. Using the method described above in connectionwith FIGS. 1A-1B, the server and workers may monitor the status of everygamelet being executed by devices of the network. As devices are addedor removed from the network, the number of workers may be changed, sothat the monitoring activities may be balanced across the workers. Whenthere is a change in status to one or more devices executing thegamelets, the change may be written to a database (e.g., database 15shown in FIG. 3) that may be accessible and/or controlled by the serverto limit the load on the database. This arrangement may provide devicemanagement for a game environment without overwhelming the server and/orany of the individual workers collecting device status.

Embodiments of the presently disclosed subject matter may be implementedin and used with a variety of component and network architectures. FIG.2 is an example computing device 10, 11 suitable for implementingembodiments of the presently disclosed subject matter. The device 10, 11may be, for example, a desktop or laptop computer, or a mobile computingdevice such as a smart phone, smart watch, smart device, tablet, or thelike, a server, networking hardware, Internet-of-Things devices ornodes, and/or any other device that may be accessed remotely and probedfor its current state.

The device 10, 11 may include a bus 21 which interconnects majorcomponents of the device 10, 11, such as a central processor 24, amemory 27 such as Random Access Memory (RAM), Read Only Memory (ROM),flash RAM, or the like, a user display 22 such as a display screen, auser input interface 26, which may include one or more controllers andassociated user input devices such as a keyboard, mouse, touch screen,and the like, a fixed storage 23 such as a hard drive, flash storage,and the like, a removable media component 25 operative to control andreceive an optical disk, flash drive, and the like, and a networkinterface 29 operable to communicate with one or more remote devices viaa suitable network connection.

The bus 21 allows data communication between the central processor 24and one or more memory components, which may include RAM, ROM, and othermemory, as previously noted. Typically RAM is the main memory into whichan operating system and application programs are loaded. A ROM or flashmemory component can contain, among other code, the Basic Input-Outputsystem (BIOS) which controls basic hardware operation such as theinteraction with peripheral components. Applications resident with thedevice 10, 11 are generally stored on and accessed via a computerreadable medium, such as a hard disk drive (e.g., fixed storage 23), anoptical drive, floppy disk, or other storage medium.

The fixed storage 23 may be integral with the device 10, 11 or may beseparate and accessed through other interfaces. The network interface 29may provide a direct connection to a remote server via a wired orwireless connection. The network interface 29 may provide suchconnection using any suitable technique and protocol as will be readilyunderstood by one of skill in the art, including digital cellulartelephone, WiFi, Bluetooth®, near-field, and the like. For example, thenetwork interface 29 may allow the computer to communicate with othercomputers via one or more local, wide-area, or other communicationnetworks, as described in further detail below.

Many other devices or components (not shown) may be connected in asimilar manner (e.g., document scanners, digital cameras and so on).Conversely, all of the components shown in FIG. 2 need not be present topractice the present disclosure. The components can be interconnected indifferent ways from that shown. The operation of a computer such as thatshown in FIG. 2 is readily known in the art and is not discussed indetail in this application. Code to implement the present disclosure canbe stored in computer-readable storage media such as one or more of thememory 27, fixed storage 23, removable media 25, or on a remote storagelocation.

FIG. 3 shows an example network arrangement according to an embodimentof the disclosed subject matter. One or more devices 10, 11, such aslocal computers, smart phones, tablet computing devices, and the likemay connect to other devices via one or more networks 7. Each device maybe a computing device as previously described. The network may be alocal network, wide-area network, the Internet, or any other suitablecommunication network or networks, and may be implemented on anysuitable platform including wired and/or wireless networks. The devicesmay communicate with one or more remote devices, such as servers 13and/or databases 15. The database 15 may be a MySQL™, PostgreSQL,Oracle™, or Spanner™ database, or the like. The remote devices may bedirectly accessible by the devices 10, 11, or one or more other devicesmay provide intermediary access such as where a server 13 providesaccess to resources stored in a database 15. The devices 10, 11 also mayaccess remote platforms 17 or services provided by remote platforms 17such as cloud computing arrangements and services. The remote platform17 may include one or more servers 13 and/or databases 15.

More generally, various embodiments of the presently disclosed subjectmatter may include or be embodied in the form of computer-implementedprocesses and apparatuses for practicing those processes. Embodimentsalso may be embodied in the form of a computer program product havingcomputer program code containing instructions embodied in non-transitoryand/or tangible media, such as floppy diskettes, CD-ROMs, hard drives,USB (universal serial bus) drives, or any other machine readable storagemedium, such that when the computer program code is loaded into andexecuted by a computer, the computer becomes an apparatus for practicingembodiments of the disclosed subject matter. Embodiments also may beembodied in the form of computer program code, for example, whetherstored in a storage medium, loaded into and/or executed by a computer,or transmitted over some transmission medium, such as over electricalwiring or cabling, through fiber optics, or via electromagneticradiation, such that when the computer program code is loaded into andexecuted by a computer, the computer becomes an apparatus for practicingembodiments of the disclosed subject matter. When implemented on ageneral-purpose microprocessor, the computer program code segmentsconfigure the microprocessor to create specific logic circuits.

In some configurations, a set of computer-readable instructions storedon a computer-readable storage medium may be implemented by ageneral-purpose processor, which may transform the general-purposeprocessor or a device containing the general-purpose processor into aspecial-purpose device configured to implement or carry out theinstructions. Embodiments may be implemented using hardware that mayinclude a processor, such as a general purpose microprocessor and/or anApplication Specific Integrated Circuit (ASIC) that embodies all or partof the techniques according to embodiments of the disclosed subjectmatter in hardware and/or firmware. The processor may be coupled tomemory, such as RAM, ROM, flash memory, a hard disk or any other devicecapable of storing electronic information. The memory may storeinstructions adapted to be executed by the processor to perform thetechniques according to embodiments of the disclosed subject matter.

The foregoing description, for purpose of explanation, has beendescribed with reference to specific embodiments. However, theillustrative discussions above are not intended to be exhaustive or tolimit embodiments of the disclosed subject matter to the precise formsdisclosed. Many modifications and variations are possible in view of theabove teachings. The embodiments were chosen and described in order toexplain the principles of embodiments of the disclosed subject matterand their practical applications, to thereby enable others skilled inthe art to utilize those embodiments as well as various embodiments withvarious modifications as may be suited to the particular usecontemplated.

1. A method comprising: assigning, at a server, a unique identifier toeach of a plurality of devices, the plurality of devices beingcommunicatively coupled to one another and to the server via acommunications network; converting, at the server, each uniqueidentifier to a device hash key by applying a hash function; splitting,at the server, a range of device hash keys of the plurality of devicesinto N approximately equal sectors, where N is a prime number and eachsector includes 1/N of the device hash keys of the plurality of thedevices; providing, at the server, K monitoring workers to monitor theplurality of devices in an order based on the respective device hashkey, where K is an integer.
 2. The method of claim 1, furthercomprising: performing, by at least one of the K monitoring workers, astatus check of one or more devices included in at least one of the Nsectors.
 3. The method of claim 2, further comprising: initiating astatus check of the next device in the one of the N sectors before thecompletion of a status check of a current device of the plurality ofdevices when it is determined that the current device is inaccessiblevia the communications network or responding below a predeterminedthreshold.
 4. The method of claim 2, further comprising: storing, at adatabase system communicatively coupled to the communications network,the determined state of the one or more devices.
 5. The method of claim4, wherein the storing comprises: storing the determined state of eachof the plurality of devices to the database system when the determinedstate has changed from a previously-stored state.
 6. The method of claim2, further comprising: determining, using one of the K workers, which ofthe N sectors to perform the status checks.
 7. The method of claim 6,wherein the N sectors range from 0 to N−1, with the j^(th) sectorstarting at hash key (2**64)/N*j+min(j,(2**64)% N) and includes(2**64)/N+j<(2**64)% N)) hash keys, wherein ** operator is a to thepower of operator, min(x,y) is the minimum of two integers, a % operatoris a remainder after division, a/operator is truncating integerdivision, and <operator is less than, which evaluates to 0 or 1, andwherein the device hash keys are 64-bit integers.
 8. The method of claim7, wherein a difference in size between a largest sector and a smallestsector of the N sectors is no greater than
 1. 9. The method of claim 6,wherein a frequency of the state checks performed by one of K workersfor each device of the plurality of devices is P, which is a measure oftime.
 10. The method of claim 9, wherein a number of K workers is from 0to K−1, and performing, at any given time t with worker number 0, thestatus checks of devices of sector number (t % (P*K))*N/(P*K), wherein %operator is a remainder after division.
 11. The method of claim 10,further comprising: performing, using workers other than worker number0, the status checks on sector number (t % ((P*i)*K))*N/((P*i)*K), wherei is the worker number between 0 and K−1.
 12. The method of claim 1,wherein at least one of the plurality of devices is a gamelet virtualmachine, the status of which is monitored by at least one of the Kworkers.
 13. A system comprising: a plurality of devices communicativelycoupled to one another via a communications network, wherein at leastone device of the plurality of devices is a hardware device; a server,communicatively coupled to the communications network, to assign aunique identifier to each of the plurality of devices, convert eachunique identifier to a device hash key by applying a hash function,split a range of device hash keys of the plurality of devices into Napproximately equal sectors, where N is a prime number and each sectorincludes 1/N of the device hash keys of the plurality of the devices,and provide K monitoring workers to monitor the plurality of devices inan order based on the respective device hash key, where K is an integer.14. The system of claim 13, wherein at least one of the K monitoringworkers performs a status check of one or more devices included in atleast one of the N sectors.
 15. The system of claim 14, furthercomprising: initiating a status check of the next device in the one ofthe N sectors before the completion of a status check of a currentdevice of the plurality of devices when it is determined that thecurrent device is inaccessible via the communications network orresponding below a predetermined threshold.
 16. The system claim 14,further comprising: a database system, communicatively coupled to thecommunications network, to store the determined state of the one or moredevices.
 17. The system of claim 16, wherein the determined state ofeach of the plurality of devices is stored in the database system whenthe determined state has changed from a previously-stored state.
 18. Thesystem of claim 14, wherein one of the K workers determines which of theN sectors to perform the status checks.
 19. The system of claim 18,wherein the N sectors range from O to N−1, with the j^(th) sectorstarting at hash key (2**64)/N*j+min (j,(2**64)% N) and includes(2**64)/N+(j<(2**64)% N)) hash keys, wherein ** operator is a to thepower of operator, min(x,y) is the minimum of two integers, a % operatoris a remainder after division, a/operator is truncating integerdivision, and <operator is less than, which evaluates to 0 or 1, andwherein the device hash keys are 64-bit integers.
 20. The system ofclaim 19, wherein a difference in size between a largest sector and asmallest sector of the N sectors is no greater than
 1. 21. The system ofclaim 18, wherein a frequency of the state checks performed by one of Kworkers for each device of the plurality of devices is P, which is ameasure of time.
 22. The system of claim 21, wherein a number of Kworkers is from 0 to K−1, and at any given time t, worker number 0performs the status checks of devices of sector number (t %(P*K))*N/(P*K), wherein % operator is a remainder after division. 23.The system of claim 22, wherein workers other than worker number 0perform the status checks on sector number (t % ((P*i)*K))*N/((P*i)*K),where i is the worker number between O and K−1.
 24. The system of claim13, wherein at least one of the plurality of devices is a gameletvirtual machine, the status of which is monitored by at least one of theK work\